Data Sources

The Data Sources page allows admins to manage all the various data sources and servers for querying and / or data modeling.

Apart from providing a convenient mechanism to add new data sources for end users (who don't need to handle all the complexities of setting up a data source), it also provides the main mechanism for governing who can see which data source (by role) and who will be able to write-back and build models into the data source where possible.

Data Sources list

To view the Data Sources list, open Pyramid and go to the Admin console (yellow arrow):

  • From the main menu, click Data > Data Source (orange arrow).

The Data Sources list opens:

Data Sources List

The following columns are displayed for all data sources

  • Actions:
  • Name
  • Type
  • Writable
  • Server IP
  • Tenant
  • Pulse
  • Encrypt
  • Description

Column View

  • Click Column View (green arrow above) to select the columns you want to show.

Custom Data Connectors

Admins are able to add their own custom data source types to the platform by uploading the necessary JDBC data drivers with the appropriate configuration. This means that users can utilize the built-in connector library or tap into the broader marketplace to add any compatible JDBC-driven data source to Pyramid at their own discretion.

  • Click here for a reference of all built in connectors.

As of version 2020.24, certain connectors are not being bundled with the main installer application for various reasons. These connectors can now be downloaded manually and added by customers from the "Connectors Marketplace" using the custom data connector framework instead. Click here for more information.

Data Sources details

When you click the Add Data Source button (purple arrow above) to create a new data source, the Add Data Source panel opens at the bottom of the page.

When you select a data source in the Data Source list, the Edit Data Source panel opens at the bottom of the page.

Adding a new Data Source Server

Click the Add Data Source button at the top-right of the page (purple arrow above) to launch the Add Data Source panel at the bottom of the page.

  • Previous and Next: Use the arrow buttons (red box, below) to move to the previous and next data cards.

Editing an existing Data Source Server

Select a data source in the Data Source list to open the Edit Data Source panel at the bottom of the page. The same dialog lets admins edit the server's role access, as well as the role access to any databases currently hosted on the data application / service.

  • Previous and Next: Use the arrow buttons (yellow arrow below) to move to the previous and next server in the data source list at the top of the page.
  • Click here for more about data source role security
  • Click here for more about pausing and resuming a data source

Common Panels

The following panel sections are shown:

General

  • Data Source Type: First, select the type of data source to add from the drop down list. The default is In Memory.
  • Display Name: Enter the user-friendly name for the data source as it appears to end users in Pyramid. Required.
  • Tenant: Next, select which tenant the data source will belong to. Each data source is specifically earmarked to each tenant in a multi-tenant role out, beyond the role settings that can be set for it.
  • Display Folder: Enter the folder name for the data source. You can create a nested (multi-levelled) folder structure by including "/" in the definition. For example "test folder/test/john" will create a nested folder structure. Optional.
  • Description: Enter the description for the data source. Optional.

The following settings are displayed for certain servers:

  • Pulse Node: this determines whether the data source will be routed through a Pulse server connection. If checked, select the relevant Pulse node to use. See below for more information.
  • Write Capable:Select to specify that the data server can be used for housing new databases. See below for more information.
  • Use 'Security and Metadata' Overlays: Select to specify that Pyramid can and should overlay metadata adjustments and / or security to the existing details from the data source. See below for more information.

Different tab headings are displayed, depending on the server chosen.

Connection

Provide the connection details to enable Pyramid to connect to the data source server. Settings vary according to server, but can include the following:

  • Server Name or IP address: Specify the host name or server address that is hosting the data application / service. Required.
  • Port: Specify the port used.
  • Optional Parameters
  • Click here for more granular information by data source type

Security

Provide the security mechanism needed to access the data source. Again, these vary according to the server chosen and can involve options like OAuth and the global account settings.

In-Memory Databases and Certificates

For In-Memory Databases (IMDB), it is possible to create a secure connection on the server-side using a private key that is in the Pyramid Certificate Manager. If you have a public+private certificate pair loaded already, you can select Encrypted (SSL) from the Secure Connection section and choose your required certificate pair from the drop-down.

Optional

These optional switches can be added to the JDBC connection string.

Finalization

Once the server details have been added, click Test in the top-right corner of the dialog. If the test is successful, click Apply to save your changes and commit the new data source to the list.

The next phase is usually setting up security on the data source in the Data Source Manager ("DSM").

  • Click Security to jump to the security panel for the data source in the DSM to setup roles and access to the server (or its child objects)
  • Click Databases to jump to the database panel for the data source to see the listing of currently materialized data on the server.

Related Capabilities

Write Capable

Certain types of data sources are write-capable. This means that Pyramid can actually use them as a DESTINATION for data that has been processed and cleansed in the Data Modeling tool. By checking this box, you allow users (with Write access) to create new databases on the targeted machine using that data stack application / service.

Note: Users need to be given explicit 'write' access to a server beyond making the data server 'write-able'.

Security and Metadata Overlay

Security and Metadata overlays in Pyramid are NOT automatically extended to MS SSAS (OLAP / Tabular)and SAP Hana. To extend this functionality, the overlay function must be enabled in the server setup page. This will provide overlaid controls for server, database and model security, as well as hierarchy and measure security and meta-data.

Note: Pyramid member level security is not provided for these engines - since these tools provide their own engines for member or row level security.

Pulse Remote Data Access

If Pulse Servers have been deployed, a remote data source can be set to use Pulse server for remote connectivity.

  • Click here for more information

Pause and Resume Servers

If your data source is moving into a period of scheduled downtime; for example, for your standard weekly maintenance procedures; you can make use of the Pause option to prevent queries to the data source being queued:

  • Click Pause to temporarily (1) prevent users from selecting or otherwise making use of the paused data source in Pyramid (to prevent future queries) and (2) kill any requests that were queuing at the point in time when the data source was paused.
  • Click Resume to return a paused data source back to its default state.

Note: If you do not pause your data source and it becomes unavailable, your queries will be queued until either the data source becomes available again or until the query times out.